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ABSTRACT 

This paper offers reasons for using traditional 
quantitative research methods and illustrates some of the 
considerations demanding rigor in a study on gender discrimination in 
school-administrator salaries. Traditional quantitative methods 
provide a meaningful language based on numbers and objectivity, and 
they are less time-consuming and complex than qualitative methods. A 
study that investigated predictors of administrators' salaries — those 
of 488 associate and assistant professors and 135 assistant 
professors — is described. Decisions made during the research-design, 
data-collection, and data-analysis stages — selecting the study 
framework, identifying relevant variables, and deciding to use an 
incremental cluster regression method — are described. The study 
concluded that gender had a significant effect on salaries, even 
after consideration of all other predictors. A conclusion is that the 
key in making decisions about research (the design, data collection, 
and data analysis) is to understand what is gained or sacrificed in 
the rigor of the study. Besides internal and external validity, 
reliability, and objectivity, an additional hallmark of rigor is 
congruency among the study* s conceptual framework, the identified 
problem, research questions, and methodology. (LMI) 
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RIGOR IN TRADITIONAL QUANTITATIVE METHODS 
Introduction 

Traditional quantitative methods are often identified with 
four major standards of rigor: internal validity, external 
validity, reliability, and objectivity. Internal validity speaks 
to the design of the study; does the study control sources of 
variance (other than that of the independent variables) that might 
influence the dependent measures, thus leading to erroneous 
conclusions about the study results? External validity speaks to 

the generalizability of the study results an issue addressed 

largely through sampling and design. Reliability speaks to the 
accuracy of measurement of the independent and dependent variables 

another possible source of error in a study, (I might add that 

instrument validity is a parallel and equally important measurement 
consideration.) Lastly, objectivity can be described as the degree 
to which the methods are value-free; that is, do the methods 
employed by the researcher bias the study results in favor of the 
researcher's predisposition? 1 

There is one additional hallmark of rigor that I would add to 

the well-recognized criteria listed above congruency between 

the study's conceptual framework, the problem identified, the 
research questions asked, and the methods employed. Although this 
standard may be so obvious that it is typically assumed to be a 
given, too few studies exhibit particularly tight congruency 
between these study elements. This lack of congruency can lead, if 
nothing else, to confusion on the part of the researcher and the 
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reader in drawing conclusion^ from the study results. 

The purpose of this presentation is two-fold: 1) to explain 
why I rely largely on traditional quantitative methods to conduct 
my research; and 2) to illustrate some of the considerations 
regarding rigor that I dealt with in conducting a study on gender 
discrimination in salaries. It is not my intent to teach you 
everything there is to know about rigor in quantitative 
methodology, for the task is far too complex and comprehensive for 
a single presentation. Rather, I hope to share with you some of 
the factors that may influence a researcher's decisions in 
conducting a particular study. 

Why Quantitative Methods? 
There are several reasons that I rely largely on traditional 
quantitative methods in conducting most of my research. First, I 

like numbers they speak to me. Numbers enable me to see 

patterns or associations that I often cannot see as well with 
verbal language. I realize that this may appear to be a rather 
silly explanation, but it is nonetheless true for me. Given all 
the recent research attention given to the importance of language, 
discourse, metaphor and its shared or individualized meaning, I 
would hope that the reader can appreciate that numbers are a 

symbolic language that has meaning for some of us often as 

much or more meaning than verbal language. 

My reliance on numbers as a meaningful language may be 
explained by the fact that I have always had a reasonably strong 
aptitude for numbers and mathematical concepts. Further, that 



aptitude was nurtured in my formal education as a math major 

in college and as a doctoral student whose research training was 
influenced largely by quantitative methodologists . As a result, I 
am comfortable with numbers and their mathematical sentences. 

Another reason that I rely largely on quantitative methods is 
that I have infinitely more patience for quantitative analysis than 
for qualitative analysis. As much as I may enjoy collecting 
qualitative data (e.g. through interviews) , I have very little 
patience for the endless sorting and prioritizing tasks that are 
required to analyze qualitative data. In a word, it makes me 
crazy! 

Perhaps the final reason I use quantitative methods to conduct 
much of my research is the objectivity that I believe it gives me. 
This is especially important to me because a good deal of my 
research addresses equity issues, including gender equity issues. 
Because I am female, I would not want my research to be biased in 
favor of women or women's issues. I feel that traditional 
posit ivist, quantitative methods grant me a degree of objectivity - 

as well as the perception of objectivity by readers that 

other methods may not. I believe that if I follow the hallmarks of 
rigor associated with quantitative methods, then the numbers (the 
results) are more likely to be credible evidence of my findings. 
Because quantitative methods have some fairly explicit guidelines 
to enhance objectivity, my findings in equity studies may less 
likely be discounted due to researcher bias in data analysis and 
interpretation . 



Addressing Rigor in the Research Process: An Illustration 

Background, Limitations, and Serendipity 

In 1988, my first study addressing gender discrimination in 
salaries of school administrators was published (see Pounder, 
1988) . Due to some of the limitations addressed in that study, I 
had decided that I wanted to do another similar piece testing the 

same conceptual model with a different occupational group 

professors of educational administration. As I was designing that 
second study, I received a survey in the mail. The survey 
represented a study of the educational administration professoriate 
conducted by McCarthy, Kuh, Newell, and Iacona et. al. (1988) in 
cooperation with the University Council on Educational 
Administration (UCEA) . 

As I completed my survey responses, I realized that the 
McCarthy et, al. instrument was measuring many of the same 
variables I had hoped to include in my salary study. I was at once 
dismayed because I felt it was unlikely that if I sent a similar 
survey instrument to the same population of respondents within the 
next two to three months that I would get a reasonable return rate. 
A weak return rate could limit the representativeness of my sample 
to the population, thus introducing a threat to the external 
validity as well as the internal validity of the study (sample 
bias) . 

At that point I decided to contact Dr. McCarthy and UCEA and 
did obtain permission to use a subset of the professoriate study 
data to conduct a salary discrimination study. Although I believe 
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I made the appropriate decision in the face of these serendipitous 
events, I was aware that a pre-existing data set has its 
limitations when adapted to a different purpose than originally 
intended. Some of these limitations may be remedied whereas others 
may not, and must be acknowledged as limitations. As my 
description of the research process unfolds, I will address the 
specific limitations imposed by the use of this pre-existing data 
set for the study of sex discrimination in compensation. 
Study Framework 

Descriptive statistics from the McCarthy et. al. data 
indicated that women faculty earned, on average, approximately 
$10,000 less per academic year than their male colleagues. When 
controlling for rank, the differential was approximately $5000 per 
academic year. Although these figures suggested gender bias in 
wage determination, descriptive statistics do not provide 
sufficient evidence of discrimination in compensation. The general 
model for determining salary discrimination is to estimate the 
portion of the salary differential attributable to relevant work, 
organizational, and market factors and to infer that the remaining 
differential is a result of discrimination in compensation 
(Milkovich, 1981) . It is important to control (typically through 
sampling or through measurement and statistical control) as many 
relevant compensable factors as possible so that the residual 
attributed to gender is not inappropriately inflated, weakening the 
validity of the study results. The work, organizational, and 
market factors generally recommended for inclusion in the model 
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are: 



A) work-related prerequisites and corequisites (e.g. 

education, training, experience) ; 

B) employee work behaviors (e.g. performance , productivity , 

absenteeism) ; 

C) work content and work responsibilities; 

D) employing organizations and industries; 

E) union membership and union characteristics; 

F) labor market conditions and regions (Milkovich, 1981) . 



Using this framework as a guideline, a review and analysis of 
related literature was conducted to identify the particular types 
of work, organizational, and market factors that were most relevant 
to a study of discrimination for this particular occupational group 

educational administration professors and to discover 

which of these compensable factors might reasonably explain a 
salary differential for males and females. The reader is invited 
to read this review and analysis (see Pounder, 1989) for a thorough 
discussion of these compensable factors and their potential 
explanatory power for the salary disparity. Briefly, however, the 
factors that were most likely to covary with gender, and thus 
influence the salary disparity were experience and job status (rank 
and tenure) , past administrative experience and/or the accompanying 
administrative salary (due to its potential effects on salary 
determination upon entry into the professorial role) , and current 
administrative responsibilities (for which there is typically 
additional compensation) . 
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Adequacy of Existing Data for Study Framework 

Because the data revealed that there were proportionately few 
women at the professor rank, I decided to use only associate and 
assistant professors for the salary disparity study (N=488) . 
(Nineteen percent of these respondents were female.) After 
examining the data set and making some recoding adjustments to more 
effectively utilize the data for the study of salary disparity, the 
following variables were available as measures of the framework's 
compensable factors : 

A) work-related prerequisites and corequisites - 

1) academic rank (associate or assistant professor) 

2) tenure status (tenured or nontenured) 

3) years of experience as a professor 

B) employee work behaviors - 

1) teaching behaviors including 

a) percentage of time spent teaching and advising 

students 

b) teaching load (standardized by semester) 

2) service measures including 

a) percentage of time spent in service activities 

b) the sum of the number of days per month spent in 

various service activites (e.g. consulting, 
attending professional meetings, guest 
lecturing, editing journals) 

3) research productivity measures including 

a) percentage of time spent in research and writing 

b) number of books written or edited 

c) number of articles , monographs, book chapters, 

and so on, written in the past five years 

C) work content and work responsibilities - 

1) administrative appointment (dean, assistant or 
associate dean, department chair, "other", or none) 

D) employing organizations and industries - 

1) type of institution (research, doctorate-granting, or 
other) 
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E) union membership and union characteristics - 

(no available data though this variable is considered 
largely irrelevant to professors as an occupational 
group) 

F) labor market conditions and regions - 

(no available data) 

Although the data set included measures for most of the 
model's compensable factors, there were some missing measures that 
warranted consideration as possible limitations to the study. 
First, an important variable identified in the analysis of the 
literature was past administrative experience and/or accompanying 
administrative salary. There was no measure of past administrative 
experience or its accompanying salary, either of which might 
influence initial salary determination upon entry into the 
professorship. Consequently, the inability to control this 
possible source of variance was identified as a limitation of the 
study. 

Additionally, there was no measure of labor market conditions 
or regions, Also, there was only one organizational variable (type 
of institution) whereas other important organizational variables 
might be size, private versus public, etc. However, the omission 
of these variables was of less consequence in a study of 
discrimination because there was no reason to believe that the sex 
of respondents would covary with these variables. Thus, the 
elimination of these variables from the model could reduce the 
total explained variance in salary, but was unlikely to 
disproportionately inflate the relationship between gender and 
salary . Because unionism among professors is fairly rare , the 
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elimination of this variable from the model was not considered to 
be a limitation of the study. 

Two other concerns with the available data existed. The first 

concern was that there was no measure of teaching effectiveness 

only teaching load and time spent in teaching. This is a common 
problem in studies of this sort because it is difficult to obtain 
a common measure of performance across different organizations. 
However, based on the review of literature, it could be assumed 
that the quality of teaching performance was randomly distributed 
across males and females with little implication for salary 
disparity. 

The final concern was one that I had not anticipated. When 
examining the descriptive statistics for the study variables, I 
noticed that the range of responses for "number of articles, 
monographs, book chapters, and so on, written in the past five 
years" was unexpectedly large (0-99) . Examination of these data 
suggested that many respondents had probably indicated the total 
number of articles that they had published, rather than those 
published within the past five years. The measurement error in 
this variable would likely attenuate its relationship with salary 
or other study variables. 

To address this problem, I decided that I would do two sets of 

parallel analyses with the study data one with associate and 

assistant professors as originally suggested (N=488) and another 
with assistant professors only (N=135) . By analyzing the data with 
assistant professors only, the measurement error in this variable 
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would be significantly reduced because their last five years of 
article publications was largely the same as their total article 
publication record. An additional advantage was that there were 
proportionately more females among assistant professors (36%) . 
However, a trade-off was that using this second sub-sample 
(assistant professors only) would restrict the range of the work- 
related prerequisite and corequisite variables, thus attenuating 
their relationship to salary or other study variables. 

In sum, the only missing variable that could likely affect the 
sex discrimination question was the past administrative experience 
and/or accompanying salary. The omission of the remaining 
variables identified above was unlikely to influence the results of 
the primary question of the study, although their omission could 
reduce the total explained salary variance. 
Data Analysis and Results 

In the interest of brevity, only highlights of the study 
analysis and results are presented here. However, the reader is 
encouraged to read the published study article (see Pounder, 1989) 
for a thorough explanation of the analysis and results. Using 
forward regression analysis, both the sample of associate and 
assistant professors as well as the sample of assistant professors 
only revealed a significant sex effect on salaries. Further, the 
slopes of the regression lines suggested that the salary 
differential due to sex discrimination in both samples was slightly 
over $3000 per academic year. 

It is important to examine more closely , however , one 
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particular portion of the data analysis. As I mentioned earlier, 
I consider one of the hallmarks of rigoro is research to be the 
congruency between the study f s conceptual framework, the problem 
identified, the research questions asked, and the methods employed. 
In conducting the regression analysis, I stumbled across a 
regression method in the SPSS manual (Statistical Package for the 
Social Sciences) with which I was not familiar, but which appeared 
to be an excellent "fit" for my study's framework. The method, 
simply referred to as "Test" in the manual, was a method for 
conducting regression analysis using clusters or combinations of 
statistically and/or conceptually related measures as predictor 
variables in the analysis. The method required a specified order 
of entry of the predictor variables so that the entry of each 
additional predictor was tested for its significance in explaining 
additional variance in salary above that of the previously entered 
variables (an incremental F-test) . 

Because my data set had several different measures for some of 
the compensable factors in the model, I decided to use this 
incremental "cluster 11 regression method to further analyze the 
data. The intent was to use the compensable factors from the model 
(i.e. their combined measures) as predictors in the regression 
analysis, thus having a tighter congruency between the model tested 
and the analysis tool utilized. Variable clusters (or compensable 
factors) were entered in the following order: 1) work-related 
prerequisites and corequisites; 2) work content and 
responsibilities; 3) type of institution; 4) work behaviors, i.e. 
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teaching, service, and research productivity; and lastly 5) gender. 
Because the purpose of the study was to test for sex 
discrimination, it was critical that gender was added last in the 
model so as not to inappropriately inflate its effects. 

Using associate and assistant professors, the analysis 
revealed that all of the variable clusters except work behaviors 
(i.e. teaching, service, and research productivity) explained a 
statistically significant portion of additional variance in 
salaries. Gender had a significant effect even after the forced 
entry of all other predictors. The parallel analysis with 
assistant professors only revealed that gender was the only 
variable that explained a statistically significant portion of 
additional variance beyond that of all other predictors previously 
entered into the regression equation. 

As anticipated, neither of these regression analyses explained 

more than 3 0% of the variance in salaries though this is not 

noticeably low compared to many studies of this type. Future 
research which includes some of the variables identified above 
could well increase the proportion of variance explained. However, 
it is doubtful that the inclusion of these variables (other than 
perhaps past administrative experience or salary) would alter the 
significant sex discrimination effect found in this study. 

Closing Comments 

I hope that I have illustrated why and how researchers make 
some of the decisions they do in the process of conducting their 
research. As I tell my doctoral students often, "There is no such 
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thing as a perfect study. 11 Trade-off decisions must be made 

constantly particularly in the design st; of research and 

sometimes, as the study above illustrates, even after data 
collection. The key in making these decisions is to understand 

what one is gaining or sacrificing in the rigor of the study 

and further, to understand what these benefits or losses will mean 
in terms of the conclusions one can draw from the study's results. 



14 



ENDNOTES 



Although these and other hallmarks of rigor are thoroughly 
addressed in many classic research methods texts, I have found 
that the Borg & Gall (1989) text is one of the more "user- 
friendly" for doctoral students conducting dissertation 
research. I would encourage students to explore more fully 
the explanations of these quantitative research 
considerations . 
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